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As the higher education landscape has expanded beyond issues of access and affordability 
to include an emphasis on student completion and employment outcomes, accreditors 
can play a leadership role in advancing this important change. A shift to student success 
that rightfully centers in part on closing equity gaps between low-income students and 
students of color and their peers ensures that students from all backgrounds have a genuine 


opportunity to thrive in and after college. 


The institutions that have progressed the most have done so through concerted, systemic, 
and equity-minded use of data to shine a light on those areas where focus and resources 
are most needed.' Indeed, for institutions today, data-use is a prerequisite to making 
institutional improvement, especially in unpacking and addressing the systemic racial and 
economic inequities that continue to undermine justice and opportunity within our higher 
education system. Accreditors, who hold primary responsibility for assuring quality and 
continuous institutional improvement, can wield enormous power in the drive to improve 


student success at more institutions by 
ACCREDITORS CAN WIELD ENORMOUS 
POWER IN THE DRIVE TO IMPROVE STUDENT 
SUCCESS AT MORE INSTITUTIONS BY USING and evaluations of colleges and universities. 


using data to shape their conversations with 


DATA. 
However, despite incremental progress, 


accreditors—primarily regional ones—presently do far too little to integrate and focus on 
quantitative outcomes data, especially data disaggregated by race and income, throughout 
the review cycle or as a basis for setting institutional improvement expectations for their 
accredited institutions. Better data-use is necessary to identify areas of success and areas in 
need of improvement, to guide institutional improvement processes, and to evaluate equity. 
Based on a review of accreditor materials and interviews with 10 high-level commission 
staff from regional, national, and programmatic accreditors, this report seeks to identify 


current practices, challenges, and opportunities with respect to data-use in accreditation. 
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We offer three recommendations for proactive steps that 
accreditors can take to incorporate outcomes-focused, 
equity-minded data into the entire review cycle to spur more 
evidence-driven institutional improvement: 


Recommendation: Embed data-use into routine practice. 
Accreditors should use data to explicitly inform their 
focus and conclusions by routinely leveraging existing 
federal data sources and, when necessary, requiring 
institutions to report additional quantitative student 
outcome data. 


Recommendation: Emphasize equity. Accreditors should 
make equity a higher priority by requiring institutions 
to report quantitative outcome metrics disaggregated 
by at least race/ethnicity, and income. 


Recommendation: Increase transparency about data-use 
practices. Building on the progress established by the 
Council of Regional Accrediting Commissions (C-RAC) 
graduation rate exercise, accreditors should increase 
transparency to the public about how they collect 
data, what data they collect, and how they use data in 
their review processes. 


In discussions with accreditors, many voiced an interest 
in using data, and this paper profiles promising ways that 
four accreditors have incorporated data into their work. 
Building on such examples, accreditors could embed data 
more thoroughly into continuous institutional improve- 
ment efforts to demonstrate their collective commitment 
to evidence-based decision making. A more thorough focus 
on data also would demonstrate to policymakers and policy 
experts that both accreditors and institutions are willing 
and able to identify and address many of the shortcom- 
ings within our higher education system. More importantly, 
when accreditors make better use of student outcome data, 
the institutions and all the students they serve can bene- 
fit from enhanced and more equitable opportunities and 
high-quality educational outcomes. 
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METHODOLOGY 


Several stages of research informed the development of 
this report. First, the authors conducted interviews with 
a dozen experts to increase our knowledge and inform our 
understanding of the historical context for accreditors’ 
data-use, the present policy and practice landscape of 
outcomes-focused accreditation, and in particular, policy- 
makers’ perceptions of data-use in quality assurance. Next, 
the research team conducted a review and analysis of 10 
regional, national, and programmatic accreditors’ data-use 
practices, examining publicly available accreditor materials 
such as standards, annual guidance and reporting require- 
ments, and a selection of institutional self-studies.” 


The core of our research findings, however, are based on 
conversations with representatives of the accrediting agen- 
cies, including regional, programmatic, and national accred- 
itors. These interviews allowed us to better understand the 
perspectives and experiences of accreditors themselves. 


To facilitate the most open and candid dialogue possible, 
the authors agreed to confer partial anonymity on inter- 
viewees; their names are listed in Appendix A, but while 
this report bases many conclusions on the results of those 
interviews and provides quotes from the interviews to sub- 
stantiate those conclusions, we have omitted attribution of 
these quotations. In instances where interviewed accredi- 
tors cited further information, the authors reviewed those 
materials. 


Prior to publication of the final version of this report, we 
convened several of the accreditors to provide them with 
the opportunity to review and voice comments about initial 
findings. 


A list of abbreviations and acronyms appears in Appendix B. 


SCOPE 


This report focuses on the issue of accreditor collection 
and use of empirical, quantitative student outcomes data, 
such as the metrics enumerated in Toward Convergence, the 
metrics framework from the Institute for Higher Education 
Policy (IHEP), which include graduation rates, retention 
rates, loan repayment rates, and others.* 


This report also includes an evaluation of what data accred- 
itors continually collect and use to assess institutional 
performance, and the barriers to and opportunities for 
improvement in such reporting and use. While these topics 
represent only a portion of accreditors’ responsibilities for 
oversight, this report demonstrates the outsized impor- 
tance that such considerations have for students, institu- 
tions, and taxpayers. 


We note also that while this report has implications for 
issues of financial and institutional sustainability, our inquiry 
did not focus explicitly upon issues of financial oversight on 
the part of accreditors, which is a substantial responsibility 
under their purview. 


Finally, while this report focuses its findings and recommen- 
dations on quantitative metrics such as measures of student 
access and persistence, graduation rates, and workforce out- 
comes of former students, we recognize that accreditation 
encompasses significant activities beyond such topics. The 
most important of the areas we do not include in our exam- 
ination is student learning. Although this is clearly a critical 
element in accreditors’ work, we do not address those indi- 
cators because at present the information that institutions 
collect on learning outcomes are difficult to compare across 
institutions, making that information less helpful for national 
and state level policy discussions, including this one.* Recent 
developments in authentic quantitative approaches to learn- 
ing outcomes assessment may hold promise for more com- 
prehensive data-use in this area in the future.® 
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Definitions 
As used in this report, the following definitions apply: 


Accreditor: There are broadly two types of accredita- 
tion in our system of higher education—institutional 
and programmatic (the latter is sometimes referred 
to as “specialized” accreditation). Institutional 
accreditation, which includes accreditation agencies 
that are either national or regional in scope, reviews 
educational institutions. Programmatic accredita- 
tion focuses on specific programs within institutions, 
such as business, engineering, law, or nursing. 


There is also a distinction between those accreditors 
that are “gatekeepers’ of federal financial aid like Pell 
Grants and federal student loans and those that are 
not. Institutional accreditation (from a national or 
regional accreditor) is required to participate in Title 
IV programs. Many institutions choose to acquire 
both institutional and programmatic accreditation 
in order to ensure the quality of their programs and 
the institution as a whole. In those instances when we 
are referring to a subset of accreditors (e.g. national, 
regional, or programmatic), we have specified it in 
the text. 


Commission representative: For simplicity and 
anonymity, when quoting the leaders of accreditation 
bodies throughout this report, we use the common 
identifier of “commission leaders’ or “commission 
representative” even though the precise title varies 
across accreditors. 


Student outcomes: Unless otherwise noted, 
student outcomes refer to quantitative metrics of 
student success, including but not limited to student 
retention, graduation rates, transfer rates, and post- 
college employment outcomes. 


KEY FINDINGS 


First, some context. In the past, accreditors have been 
maligned as only focusing on weak input measures—for 
example, the number of volumes in institutions’ libraries— 
but that is an incomplete and unfair characterization. 
Accreditation review, because it is based in peer review, 
has been historically driven by qualitative indicators, judge- 
ments founded in peers’ professional experience. Now that 
the higher education field has more fully embraced quanti- 
tative indicators as well, it is possible for these to be com- 
bined with accreditors’ focus on qualitative indicators to 
provide more meaningful, comprehensive, and proactive 
quality assurance. 


One example of increased attention to identifying mean- 
ingful quantitative indicators is the 2018 review of institu- 
tions with low graduation rates by the Council of Regional 
Accrediting Commissions (C-RAC)—a group of regional 
accreditors themselves.° Some accreditors are working 
to go further, designing their own measures and methods 
of incorporating outcome data to inform their reviews, 
and a few accreditors (especially those that oversee 
career-focused programs) make accreditation decisions 
based in part on outcome metrics, some of which include 
thresholds at which specific consequences apply. 


Against this landscape of accreditation’s history and these 
recent developments, this section describes key findings 
from our research. 


Finding: Accreditors recognize the value of improving the 
availability, uniformity, accuracy, and timeliness of 
federal postsecondary data collections. 


The benefits of making more data available. In interviews, 
nearly all of the commission leaders recognized the value of 
increasing the quality and comprehensiveness of data made 
available through federal collections. For example, when 
discussing new Integrated Postsecondary Education Data 
System (IPEDS) Outcome Measures (OM), a graduation rate 
metric introduced by U.S. Department of Education (ED) in 
2017 that accounts for transfer and part-time students, one 
commission representative said, “We're very pleased that 
they‘re [ED] doing that [OM]. | think everyone is quite happy 
about that.” This comment was echoed in many of our other 
conversations. 


These comments aligned with one of the primary points of 
consensus established in the C-RAC graduation rate study: 
acknowledgement of the importance of using data and, 
specifically, a shared commitment to improving accreditor 
use of data. In that report, regional accreditors collectively 
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What do we mean by “data-use”? 


When referring to “use” of data, we are broadly refer- 
ring to two distinct but related actions. First, we are 
describing a comprehensive approach to integrating 
quantitative data on student outcomes throughout 
each phase of the accreditation process. Specifically, 
once a group of accreditors receives data from an 
institution, data-use would include, at a minimum: 
(1) an initial analysis of data to inform the structure 
and content of site-team accreditation reviews; and 
(2) during the accreditation process, ongoing assess- 
ment of outcomes, as revealed through data, used to 
inform conversations, conclusions, and decisions. 
This type of data-use would enable accreditors to 
direct limited staff resources and time to institutions 
evincing the most concerning results on quantitative 
metrics, allow them to prospectively develop site vis- 
its and requests for further written information based 
on each institution's strengths and weaknesses as 
revealed in quantitative outcomes, and, finally, to 
implement collaborative improvement plans that 
set realistic but aspirational quantitative goals for 
improvement. 


Second, “data-use” may also refer to imposing conse- 
quences based on failure to meet a threshold or com- 
ply with a set of metrics. These consequences could 
include probation, orders to show cause, or more 
severe actions such as revocation of an institution's 
or program's accreditation. In addition, these con- 
sequences can include constructive steps: further 
investigation into the factors driving performance on 
specific metrics, or additional accreditor support for 
institutional improvement in key areas. 


Because we frequently are referring to data-use in 
only one of these senses, to delineate between the 
two, we refer in the text either to integrating data 
into the review process or imposing consequences, 
respectively. In no instance do we use the term to 
mean only developing bright-line indicators that are 
used to revoke accreditation. 


NEARLY ALL COMMISSION LEADERS SAID THAT A TOP 
PRIORITY WAS GETTING ACCESS TO BETTER DATA, 
ESPECIALLY FROM A TRUSTED PUBLIC SOURCE LIKE 
THE FEDERAL GOVERNMENT. 


expressed that they were “fully committed to helping their 
regions and individual institutions improve their graduation 
rates and to help policymakers in their efforts to improve 
graduation rate measures and hold institutions account- 
able.” The study reported that the same held true for institu- 
tions, who recognized that “there is a significant amount of 
effort” in reporting these figures, and told accreditors that 
“they do prefer it, that they think they get real value out of 
knowing how they’re performing relative to these bench- 
marks and they can use those benchmarks to set goals.”” 


The need for uniformity across institutions. In addition to 
acknowledging the usefulness of data generally, most of 
the commission leaders also sought a common typology of 
measures and terms, so that data are not only more straight- 
forward to calculate but are more comparable across insti- 
tutions and programs. They wanted improvements including 
common, validated definitions of metrics and the ability to 
report outcomes at various points in students’ educational 
trajectory. For example, one commission representative 
said, “I'd love to just come up with a common definition of 
graduation, what is ‘graduated’? If we could do that, that 
would solve a lot of problems, but everybody wants to 
exclude every student from the graduation rate calculation.” 
Despite these limitations, graduation rates are commonly 
used by accreditors, but improved federal data could also 
include better measures of less commonly used factors, 
such as transfer rates or post-college outcomes. 


The value of ensuring data accuracy and timeliness. In our 
interviews, nearly all commission leaders said that a top 
priority was getting access to better data, especially from 
a trusted public source like the federal government which 
regularly provides data on key measures through sources 
such as IPEDS, the College Scorecard, and others. These 
federal data sources contain limitations and are not fully 
comprehensive, but they are relatively consistent and reli- 
able. Federal data are also available without a fee. 


At least one commission leader acknowledged that some of 
the existing data are only provided at the state level, which 
is insufficient, since with “state data systems, if it's not con- 
nected to anything else really, you get very incomplete data.” 
Others recognized the value of the federal government's 
role in providing more complete information on student 
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outcomes, saying: “| think the [student-level data network ]é 
would help. It's not going to do the whole thing, but | cer- 
tainly do think it would help.” Another said, “The data | wish 
| had was what happens to students who drop out. There's 
just nowhere to know where they get picked up again, or 
what happens to them. Do they reappear somewhere? That's 
a challenge.” This lack of access to important data is a real 
barrier to institutional quality improvement and accreditor 
oversight. One commission representative stated, “If there 
were data in which we felt sufficiently confident, we might 
have a bright line minimum number or a regulatory release 
sort of thing that followed from a top line that we thought 
was good enough. Problem is, we don't have them.” 


Finding: Accreditors collect several institution-level 
measures of student access and success. 


Progress on incorporating data—but quality and metrics 
vary across accreditors. A review of accreditors’ standards, 
institutional self-studies, and annual requests for additional 
information demonstrates that all accreditors are requir- 
ing institutional reporting of several quantitative outcome 
metrics. In our interviews, most commission leaders agreed 
that over the last couple of years, “there is an increasing 
emphasis on using data and an increasing attention to aca- 
demic quality and academic success, student success.” 
Many commented that, thanks in part to improved technol- 
ogy, their teams are improving their ability to collect data, 
and to find data useful. 


The choice of metrics, however, varies—sometimes 
considerably—across agencies (see Table 1 for specific data 
collected by each participating accreditor). And it is unclear 
the extent to which all accreditors use the collected data—of 
whatever metrics—to drive colleges to improve their perfor- 
mance, which we discuss in more detail below. 


Accreditors recognize that more can be done, including 
improving the quality of data reported by institutions. One 
commission representative said, “The use of data is a hot 
topic even at the commission level, and we have a data 
working-group of staff and commissioners really trying 
to peel this exact onion of, what are the data elements we 
need... as well as what are the key data elements that from 
our policy perspective are most useful to the team mem- 
bers, to the staff in understanding what's happening at the 
institution as well as making accreditation decisions.” 


Examples of data collection. Here are other ways accredi- 
tors are collecting data, supplementing existing data collec- 
tions, and using data in reviews: 


e 


Including data from multiple sources. Accreditors are 
making efforts to collect data from multiple sources, 
including IPEDS and the National Student Clearinghouse. 


Increasing the number of data points. Over the past sev- 
eral years, accreditors have increased the number of data 
points collected to provide a more complete picture of 
changes in performance. 


Exploring new metrics and a multiple measures approach. 
Under the title of the Graduation Rate Dashboard,® 
the WASC Senior College and University Consortium 
(WSCUC) developed new metrics for graduation and credit 


Table 1: Overview of accreditor data collection 


redemption, intended to supplement the federal gradua- 
tion rate definition by using multiple measures. 


Including employment outcomes data. There is growing 
commitment, primarily among accreditors overseeing 
career training programs, to collecting employment out- 
comes data, including the use of employer surveys.” 


Exploring options for measurements through newresearch. 
Emerging accreditation-focused research commissioned 
by the Higher Learning Commission (HLC) and conducted 
by its member institutions and other stakeholders has 
highlighted topics such as ways to holistically measure 
student success, including accounting for varying stu- 
dent goals and risk factors, and addressing how data can 
effectively drive improvement. 


Summarized below is an examination of whether the accreditors reviewed in this report collect various quantitative outcome 
metrics, and whether those metrics are disaggregated on the basis of race, Pell Grant status, or both. We have selected the 
most critical metrics articulated in IHEP’s “Toward Convergence” metrics framework, and based our initial research into these 
metrics on the analysis in the Center on American Progress (CAP) report on accreditor data collection and use. This analysis 
evaluates the collection of these data based on publicly available accreditor information, primarily annual information collections, 
supplemented in some cases by a sample of institutional self-study documents. Prior to publication of the final version of this 


report, the accreditors listed were given an opportunity to review and voice comments and concerns, if any. 


Total enrollment e e | Ea e e e e e 
Number of completers e e e e e e e 

Completion rate e e e e Sim 
Cohort default rate e e e e e e e 
Retention/withdrawal rate e e e e e 
Licensure/certification passage rate e e e e e 

Transfer-out rate e e e e e 


Loan repayment rate 


Employment rate (for career programs) e e 


Median earnings e 


Credit completion 


Credit accumulation 


Gateway course completion 


mu Elements disaggregated by race and Pell status 


Ww 


Elements disaggregated by race only 


" Elements disaggregated by Pell status only 


NWCCU does not require disaggregated reporting on the basis of income or race, but does require the institution to report whether they are designated by ED as one 


or more classifications of Minority Serving Institutions. 


ACCSC requires disaggregated reporting of enrollment on the basis of Pell receipt and ethnicity, rather than race. 

NECHE is the only accreditor that we reviewed that explicitly requested IPEDS OM measures in addition to their own graduation rate measure. 

SACSCOC collects completion rate data in the form of IPEDS Graduation Rate, IPEDS Outcomes Measure, and National Student Clearinghouse Total Completion Rate; 
for those institutions that do not report to IPEDS, institutional data is provided directly to SACSCOC using IPEDS completion formulas. SACSCOC disaggregates 


completion rate on the basis of ethnicity, rather than race. 
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e Introducing new data collection tools. Many also are start- 
ing to use more robust data tools, including Salesforce. 
One commission representative stated that, “We're mov- 
ing to using Salesforce as a data management system. | 
was just in a meeting with the presidents of the other 
regionals. It looks like almost all of us are going to use 
Salesforce which would give us a common database, a 
way to share data.” 


Financial advantages to increased data-use. There is also 
accreditor recognition of the potential—which has, to some 
extent, already been demonstrated—to lower the cost and 
effort of making improvements by using data to diagnose 
problem areas. One commission representative pointed to 
lack of resources and staff as a reason to increase use of 
empirical data to drive decisions: “You know, we're small 
nonprofits so there's only so much we can do, but the entire 
universe has improved so much in the last five years that 
we've been able to do more than we ever could from a bud- 
getary point of view and just from a human resources point 
of view.” 


Finally, there was an emerging recognition among several of 
the commission leaders that were interviewed that better 
use of datais critical not only to the success of institutions, 
but ultimately to their survival as well. As one commission 
representative acknowledged, there are “enormous practi- 
cal incentives for institutions to do this [given the] declining 
population of 18-year-olds. So, institutions that want good 
enrollment statistics have every incentive to work on this. 
So, it's not just an educational motivation or moral motiva- 
tion or any of that. It’s also really a financial motivation.” 


Finding: Accreditors repeatedly refer to “using” data in 
reviews, but there is little evidence that many accred- 
itors integrate data into the review process or base 
consequences on data. 


Differences among regional, programmatic, and national 
accreditors. The recent effort of regional accreditors, 
through C-RAC, to identify and work to improve institu- 
tions with low graduation rates is an important example 
of accreditors’ progress on using data—which resulted in a 
study and report of regional accreditors’ analysis of gradu- 
ation rate data. Through that study, accreditors identified 
institutions that needed immediate accreditor involvement. 


The report's findings also reflected how seldom accredi- 
tors leverage the data that are collected. For example, while 
many accreditors have long collected graduation rate data, 
the C-RAC report spurred them to ask their low-performing 
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IT'S NOT JUST AN EDUCATIONAL MOTIVATION 
OR MORAL MOTIVATION OR ANY OF THAT. 
IT'S ALSO REALLY A FINANCIAL MOTIVATION. 


institutions for plans on how to improve, with at least one 
accreditor requiring additional reporting and justification 
from several institutions with low graduation rates, in one 
instance feeling that issues identified through the data jus- 
tified a site visit. Without these C-RAC-inspired ongoing 
evaluations, there is astrong possibility that these problems 
would not have been proactively identified. 


Although most accreditors say they use data frequently, 
when pressed on what that data-use looks like, regional 
accreditors often did not provide evidence that data drives 
specific decisions regarding accreditation or improvement 
requirements, outside of this recent C-RAC effort. 


One programmatic commission leader straightforwardly 
agreed with this conclusion in terms of accreditation prac- 
tices, stating that “We're probably using less data that you 
might have imagined.” This commission representative 
seemed to signal there was indeed some level of low per- 
formance that would signal negative consequences, though 
it is unclear what such consequences would be: “Reporting 
that [outcome] data does not influence our accreditation 
decisions directly. Although, let me just give you a caveat. If 
it turned out an institution, a program that we went to visit, 
was graduating less than 5 percent of the students, it would 
be an issue for us.” 


Other descriptions of data collection and use that many 
accreditors shared during interviews were nonspecific: 
there were repeated mentions of “exploring” predictive 
analytics, conducting additional research, and “supporting 
improvements’ at institutions. 


Our interviews and a review of underlying accreditor doc- 
umentation confirms what previous research found"—the 
national accreditors we reviewed are further along not only in 
evaluating the data they collect but in using those data to set 
benchmarks and in some cases to hold institutions account- 
able when they fall short. In fairness, national accreditors 
have relatively fewer barriers to implementing such a system, 
both because of the relative homogeneity of their institu- 
tions (as compared to regional accreditors) and the predom- 
inantly career-focused nature of the educational offerings, 
which make it easier to Compare measures such as earnings, 
employment, and licensure passage rates. However, national 
accreditors—such as the Accrediting Commission of Career 
Schools and Colleges (ACCSC) have still put in the effort 
necessary to identify and define these measures, including 


DATA SHOULD BE USED TO DESIGN OR FOCUS 
TECHNICAL ASSISTANCE, IDENTIFY HOW TO BETTER 
ALLOCATE RESOURCES, AND CONNECT INSTITUTIONS 
WITH PEERS FOR ASSISTANCE AND LESSONS LEARNED. 


requiring surveys and audits in some cases(see more detail in 
the Case Studies sidebox). Those efforts could be replicated 
or adapted to the regional and programmatic contexts. 


Improvement and consequences. Improvement and con- 
sequences should be closely linked. But, significantly, we 
have seen no compelling evidence that institutions that do 
not improve on student outcome measures are at risk of real 
consequences, like loss of accreditation. For example, the 
C-RAC report noted that when HLC conducted a study that 
assessed what various institutions do to improve graduation 
rates, they found that institutions “Monitor course comple- 
tion rates of their students; Monitor the transfer-out rate of 
students; Set a target graduation rate; Monitor the gradua- 
tion rate of students who are not included in the IPEDS grad- 
uation rate reported to the U.S. Department of Education.” 
Another regional commission representative said, “We will 
see how well they're doing, recognizing you don’t change 
graduation rates overnight. But to at least make sure they're 
at least making progress in moving forward.” 


This monitoring is a necessary prerequisite to encouraging 
appropriate institutional responses but monitoring alone is 
insufficient to ensure institutions are making continuous 
improvements for all students, particularly low-income stu- 
dents and students of color. In this sense, data should be 
used to design or focus technical assistance, identify how 
to better allocate resources, and connect institutions with 
peers for assistance and lessons learned. 


Our study found that some accreditors—both regional and 
national—are making a more explicit commitment to using 
data to set benchmarks that would, at minimum, result in 
this type of targeted continuous improvement in response 
to institution or program-level performance issues. For 
example, the Southern Association of Colleges and Schools 
Commission on Colleges(SACSCOC) is attempting to bench- 
mark its diverse institutions by having institutions them- 
selves set benchmarks and comparison peer groups—with 
initially encouraging results (see more detail in the Case 
Studies sidebox). 


For those accreditors that do set benchmarks, commission 
leaders emphasized that a failure on a benchmark does not 
serve as a basis to revoke accreditation; instead, a drop or 
a failure on a metric begins a collaborative evaluation and 
improvement process. One commission leader said, “don't 
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get the impression that if a program dips below the bench- 
mark, it’s automatically done. It’s the beginning of an anal- 
ysis and sort of beginning of the conversation around, all 
right, what's going on with this program?” By the same token, 
however, reaching the benchmark is not the end of the over- 
sight evaluation. The same commission leader expressed 
this sentiment, noting that “the benchmark is not your goal. 
If you're just shooting for the benchmark, congratulations, 
you're below average.” 


Another commission representative offered a different 
reflection on tying consequences to data, saying “Just like 
any other standard, when they’re out of compliance, they're 
out of compliance. At some point, it could end up that they’d 
be on warning or probation or drop from membership.... 
We don’t have though, what | would call or what Margaret 
Spellings used to call ‘bright line indicators,’ for graduation 
rates or anything else.” 


Finding: Accreditors very rarely disaggregate data for pur- 
poses of promoting racial, ethnic, and socioeconomic 
equity. 


Our higher education system continues to perpetuate 
racial and socioeconomic inequities in college enrollment 
and completion.” Yet, some institutions have narrowed or 
closed equity gaps, promoting high levels of success for 
all students and demonstrating that what institutions do 
matters. It is also true that, as arbiters of quality at colleges 
and universities, what accreditors do matters in ensuring 
that students of color and students from low-income back- 
grounds benefit from high levels of postsecondary quality— 
just as their white and wealthier counterparts do. 


In our interviews and research, however, we found that apart 
from a few instances, accreditors do not require data disag- 
gregation by race or income nor do they focus explicitly on 
equity. The sole metric we found to be collected ona disaggre- 
gated basis by multiple accreditors was enrollment. Only one 
accreditor collected disaggregated data on graduation rates 
on the basis of both race and Pell status, while one accredi- 
tor disaggregated graduation rates and the number of com- 
pleters for Pell status only. Finally, one accreditor looked at 
loan repayment rates disaggregated by Pell status (see Table 
1). Accreditors otherwise do not collect or require disaggre- 
gated data by race or income, even though disaggregated 
leading indicator metrics, such as retention, credit accumu- 
lation, or gateway course completion could be especially use- 
ful for continuous improvement efforts because they would 
allow accreditors and institutions to take action in real time, 
helping students while they are still enrolled and identifying 
and addressing equity problems as early as possible. 


Case Studies 


Some accreditors are proactively using data in innova- 
tive and productive ways. Not all practices will be uni- 
versally applicable in separate accreditation contexts, 
however, and it is unrealistic to assume that all accred- 
itors could undertake identical initiatives. These exam- 
ples illustrate a few ways accreditors have demonstrated 
not only a commitment to the importance of data-use, 
but the ability to overcome shortcomings of existing 
systems by proactively making voluntary improvements. 
Accreditors have opportunities in several contexts— 
including the Council for Higher Education Accreditation 
(CHEA) and C-RAC—to share promising practices and 
learn from one another, and we recommend that accred- 
itors leverage these chances for collaboration to make 
improvements to their existing processes. 


SACSCOC measures progress against baseline 
performance on completion metrics. 


SACSCOC has addressed the challenge of diverse insti- 
tutional missions and student populations by offering 
institutions several completion metrics from which to 
choose. In 2017-18, member institutions with under- 
graduate programs were asked to identify a key student 
completion indicator from the following completion met- 
rics: (1) the completion rate based on the data annually 
reported to SACSCOC by member institutions, (2) the 
“traditional” IPEDS overall graduation rate (within 150 
percent of time), (3)the IPEDS Outcome Measures(8-year 
award rate), and (4) the National Student Clearinghouse 
total completion rate (6 years). The institution’s perfor- 
mance on that year’s selected key student completion 
indicator was used to create a baseline performance 
level, and subsequent performances were compared to 
baseline levels. 


At that time, the Commission also asked institutions to 
select approximately 10 institutions they considered to 
be their peers in the region. As aresult, the Commission 
was able to provide each member institution with its own 
performance data using their preferred completion indi- 
cator, along with the average performance of their peers 
on that metric. Institutions were then asked to include a 
discussion of student success dynamics on the selected 
key completion indicator in the decennial Compliance 
Certification Report and in the Fifth-Year Interim Report. 
Ongoing peer evaluation committees have used this 
information as context to inform their reviews. 
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ACCSC defines completion and employment 
metrics and thresholds. 


ACCSC relies on a variety of information when reviewing 
institutions and programs. More specifically, since 1998, 
ACCSC has focused their student achievement metrics 
on graduation rates and employment rates using data 
they collect outside of federally provided data. ACCSC 
has calculated its graduation and employment bench- 
mark rates based on cohorts of students who start in the 
same program at the same time. Graduation is measured 
at 150 percent of normal time to program completion, 
and the employment rate measure three months after 
graduation to allow for students to secure a job. 


Once every cohort's graduation and employment rates are 
calculated, the data is aggregated to establish a gradua- 
tion rate and employment rate for each program. Those 
rates are then evaluated relative to ACCSC’s published 
graduation and employment benchmarks. Graduation 
rate benchmarks vary based on program length and are 
set one standard deviation below the average gradu- 
ation rate for programs of similar length. The employ- 
ment rate benchmark is set for all program regardless 
of length because the correlation of a program's length 
to its employment rate is not as great as it is for grad- 
uation rates. For example, the established benchmark 
graduation rates for programs 1-3 months is 84 percent, 
for programs 10-12 months is 55 percent, for programs 
19-23 months is 43 percent, and for program 24 months 
and greater is 40 percent. The established employment 
rate benchmark is 70 percent for all programs. 


Every year ACCSC members submit graduation and 
employment data, allowing ACCSC to assess perfor- 
mance and evaluate what actions, if any, need to be taken 
to press quality vis-a-vis student achievement. When 
a school or program approaches a benchmark “danger 
zone” this sets off a figurative alarm; at that moment, 
ACCSC may require more information, detailed report- 
ing, or heightened monitoring. Later repercussions could 
include an on-site visit, and if the program still does not 
improve, the Commission would likely revoke a program's 
approval or take an institutional action such as a Warning 
or Probation. 


continued on page 12 > 


Case Studies, continued from page 11 


ACEN benchmarks program performance on 
completion, job placement, and licensure pass rates. 


As an accreditor for nursing programs at a wide vari- 
ety of degree levels, the Accreditation Commission for 
Education in Nursing (ACEN) emphasizes continuous 
self-assessment and focuses on several outcome mea- 
sures. These measures include program completion, job 
placement, and licensure examination pass rates—the 
latter of which is assessed at the state level. 


In regard to program completion and job placement, 
ACEN requires institutions and programs to set their 
own benchmarks of success using measurable, realistic, 
and reasonable expected levels of achievement (ELA) 
for completion and job placement rates. In each annual 
report, institutions and programs must identify their 
assessment methods and specify how their performance 
compared with the ELA. While institutions and programs 
set their own ELA benchmarks, ACEN is involved in that 
process and provides encouragement and support to 
nudge programs towards continuous improvement. 


For licensure examination pass rates, each State Board 
of Nursing regulates nursing standards and sets the pass 
rate for nursing programs in their respective states. 
ACEN decided to adopt 80 percent, the most common 
pass rate set across a majority of states, as their bench- 
mark. Once licensure exam information is published for a 
given year, the institution and program must report that 
data and assess how well they performed in relation to 
the 80 percent benchmark. 


Overall, and arguably most importantly, institutions and 
programs must collect assessment data to measure 
against these expected levels of achievement and licen- 
sure pass rates, analyze that measurement, and provide 
documentation demonstrating that the assessment 
data are being used to maintain and improve student 
outcomes. 


WSCUC creates a graduation rate dashboard 
(GRD) metric and embeds key indicators in the 
institutional review materials. 


WSCUC emphasizes “multiple measures,” including the 
Graduation Rate Dashboard (GRD) it developed to com- 
plement and fill in some of the deficits that existed in 
2014 in IPEDS. The GRD sought to build upon existing 
first-time, full-time federal graduation rates and aims 
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to do so in a way that counts all students regardless of 
how they enrolled (first-time or transfer, lower or upper 
division, part-time, full-time, and for all types of degree 
programs). 


The GRD collects six data elements to quantify both the 
Unit Redemption Rate (URR) which measures the pro- 
portion of credits that can be ascribed to a particular 
institution toward a student's degree completion, and 
the Absolute Graduation Rate (AGR) which uses the URR 
to develop an estimate of the proportion of entering stu- 
dents at an institution who successfully graduate regard- 
less of how long it takes them. One benefit to using URR 
is that it is not dependent ona fixed timeline and tends to 
be more sensitive to how much time, effort, and money 
students invest in an institution prior to dropping out. 


WSCUC requires schools to “engage” with the GRD, 
meaning institutions must be able to have a meaningful 
discussion about GRD and any other outcomes that mat- 
ter to them, how they measure those outcomes, and how 
they connect outcome data and other indicators to stu- 
dent success improvement. 


WSCUC has also started to present a few “key indicators” 
to lead off its internal institutional review materials, so 
that at every stage the review team, decision-makers, 
and staff are attuned to basic performance metrics and 
context. The current data points available across the 
agency's universe of institutions are graduation rates 
(4- and 6-year IPEDS plus the GRD), overall graduate and 
undergraduate enrollment and percent Pell enrollment, 
cohort default rate, total expenditures, and federal 
financial composite score. WSCUC uses these measures 
to call attention to outcomes in preparing lines of inquiry 
for discussion with institutions and to identify areas of 
possible concern or effectiveness. This opens the door 
for schools to share and reflect on additional data they 
find most revealing and useful for understanding where 
they are and whether their efforts are making a differ- 
ence. WSCUC’s next steps include (1) building out trend 
and then comparative information to deepen the conver- 
sation, (2) continuing to prepare site evaluators and staff 
to be consistent and skilled in guiding effective explo- 
ration of outcomes relative to accreditation standards, 
and(3)incorporating existing student learning outcomes 
measures and identifying additional high-value metrics, 
especially for post-graduate success, to include in this 
process. 


These findings about the lack of data disaggregation were 
reinforced in our conversations with accreditors. One com- 
mission representative said that while the accreditor’s staff 
evaluates data disaggregated “by Pell status, we have not yet 
looked at the other demographic information. We'll look at 
HBCU [historically black colleges and universities ] rates ver- 
sus PWI [primarily white institutions] rates. But otherwise, 
we don't use the racial or gender data. We haven't.” Other 
commission leaders were more blunt in their assessments 
of data disaggregation. One said, “we do not disaggregate 
data” based on race, ethnicity, income, or disability status; 
another said, “we don't really get into... we don’t ask for 
demographics.” Finally, a third commission representative 
commented, “this [improving results for historically under- 
served students] is not the focus of accreditation. | can’t 
count the number of times | have heard ‘if only accreditation 
agencies required X.’ The agenda for accreditation agencies 
is improving educational quality for all and to be a reliable 
source for judging educational quality. That is our charge by 
the federal government and this must be our focus. If we fail 
in our charge, then we won't exist and then everyone loses. 
Accreditation isn't here to solve social issues.” 


AS ARBITERS OF QUALITY AT COLLEGES AND 
UNIVERSITIES, WHAT ACCREDITORS DO MATTERS IN 
ENSURING THAT STUDENTS OF COLOR AND STUDENTS 
FROM LOW-INCOME BACKGROUNDS BENEFIT FROM HIGH 
LEVELS OF POSTSECONDARY QUALITY—JUST AS THEIR 
WHITE AND WEALTHIER COUNTERPARTS DO. 


Our review of standards and public documents underscored 
these comments, showing that many accreditors have not 
demonstrated a public commitment to addressing equity 
considerations by incorporating equity or data disaggre- 
gation requirements into their standards and policies (see 
the Sidebox on Equity, Diversity, and Inclusion Policies for 
notable exceptions). 


Although data disaggregation is not enough, in and of 
itself, to focus institutional resources and commitment 
toward students who have been historically underserved 
by higher education, it is a necessary precondition to such 
efforts. Without such data disaggregation, institutions and 
programs will not be aware of access and success gaps 
between their students of color and white students, much 
less will they be able to evaluate whether interventions to 
close such gaps are effective over time. 
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Equity, Diversity, and Inclusion Policies 


Only two regional accreditors explicitly mention equity 
or disaggregation in their standards and policies, and 
only a few nod to the related issues of diversity and 
inclusion. Those who do reference equity include 
WSCUC, whose “equity and inclusion policy” states 
that a “commitment to student learning and success 
requires that institutions actively seek to support the 
success of all of their students” and that institutions 
must demonstrate “the willingness and capacity to 
identify and address equity concerns among cam- 
pus constituents.”* WSCUC also evaluates equity by 
requiring institutions to compare academic success 
of subgroups of students, and evaluates inclusion by 
requiring institutions to measure general student sat- 
isfaction and campus climate. 


Similarly, the Accrediting Commission for Community 
and Junior Colleges (ACCJC) requires that each insti- 
tution “disaggregate and analyze learning outcomes 
and achievement for subpopulations of students. 
When the institution identifies performance gaps, it 
implements strategies, which may include allocation 
or reallocation of human, fiscal and other resources, 
to mitigate those gaps and evaluates the efficacy of 
those strategies.” ACCUC also requires that institu- 
tions’ educational methods must “reflect the diverse 
and changing needs of its students, in support of 
equity in success for all students.” 


Other accreditors do not address equity in outcomes, 
but do have standards that signal the importance of 
diversity and inclusion. For instance, HLC requires 
that “the institution's processes and activities reflect 
attention to human diversity as appropriate within 
its mission and for the constituencies it serves”® 
and that “the institution engages with its... constit- 
uencies and communities of interest and responds 
to their needs.” The New England Commission of 
Higher Education (NECHE) requires that “the institu- 
tion addresses its own goals for the achievement of 
diversity among its students and provides a safe envi- 
ronment that fosters the intellectual and personal 
development of its students.”® 


Finding: Accreditors face real but surmountable obstacles 
that impede progress on improving data collection 
and use. 


The realities of our higher education system and limits 
around federal data collection create barriers to systematic 
and continuous accreditor use of data—barriers that may 
become more complex as new types of programs become 
more common, such as very short-term programs, direct 
assessment (credit for prior learning), competency-based 
assessment, and subscription models, which allow students 
to use self-directed study to complete course or credential 
requirements on anon-regularized schedule. 


These issues are real, but they do not represent insurmount- 
able barriers that prevent improved data collection and use. 
Barriers cited by accreditors—and attempts to address 
them—include: 


Diversity of institutions and programs. Many accreditors 
oversee a wide range of institutions and programs, creat- 
ing complexities in developing robust data-use frameworks, 
and most commission leaders noted that this variation was 
the most significant barrier to their improved collection 
and use of data. Even among accreditors that predomi- 
nantly oversee a single type of institution(such as four-year 
degree granting institutions), those institutions may vary 
widely in their programmatic offerings, including gradu- 
ate studies, shorter term or career-oriented programs, or 
locally-focused programs. One commission representative 
said, “We do not use a placement rate, because that’s just 
not going to work when most of your students and programs 
are for working professionals like our schools.” 


National and programmatic accreditors, however, have a 
relative advantage over regional accreditors when it comes 
to standardizing metrics across institutions. Because their 
member institutions are typically more uniform in their 
design and student profile, it is easier to define common 
metrics and set benchmarks for comparison. 


At our convening, some commission leaders suggested that 
accreditors with just a few institutions of a particular type 
could collaborate with other agencies who have more insti- 
tutions in that category to get a fuller comparison group 
and/or more context for comparative performance. 


Improving, but still incomplete nature of first-time, full- 
time graduation rates. For years, due to shortcomings of 
the federal definition of graduation rate, the only federally 
available data on graduation rates was limited to first-time, 
full-time students, representing only about half of today’s 
college-goers." 
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The C-RAC report based a significant portion of its findings 
on the problems with that metric for institutions with high 
rates of transfer. But while the report included some dis- 
cussion of the new IPEDS Outcome Measures (OM), which 
addresses a significant number—albeit not all?°-—of the 
measurement issues and is already freely available to the 
public, it does not provide justification for why OM should 
not become one key method of measuring completion in the 
near term. Indeed, some institutions and accreditors have 
already addressed some of these shortcomings by looking 
at trend analyses, multi-year averages, multiple measures, 
or a variety of data sources. And, in more thorough conver- 
sations with accreditors, they indicated less dire concerns 
about the limitations of the federal graduation rate mea- 
sure. During our convening, many accreditors stated that, 
while available metrics still have limitations, such as a lack 
of shorter term outcomes and data on what types of insti- 
tutions students transfer to, they are less concerned today 
about having to work around the limitations of first-time, 
full-time graduation rates than they have been in the past. 
More broadly though, the fact remains that well-crafted and 
valid measures require more thought and effort than simply 
flipping a switch labeled “better data.” 


Furthermore, the addition of OM to IPEDS shows that fed- 
eral data collections can be improved in response to field 
pressure. Accreditors can be strong advocates for specific 
improvements to federal data systems to facilitate their 
own data-use practices—and should continue to identify 
what is needed for the field and advocate to policymakers 
for improvements. 


Lagging outcome metrics. Graduation-rate measures are 
retrospective, lagging indicators and do not demonstrate 
more recent institutional changes. Institutions that would 
be considered as having “low” graduation rates can shift 
significantly from year to year and so any single-year mea- 
sure risks overreaction. Conversely, institutions that may 
have evinced no warning signs for several years may have 
sudden negative changes that would not be reflected in 
averages over time, so both averages and sudden changes 
should be considering for a holistic institutional view. To 
give credit to institutions that are currently improving, sev- 
eral commission leaders suggested supplementing lagging 
indicators with others that are more immediate and can 
provide context—like retention or gateway course comple- 
tion. One commission representative said in our interview 
“Graduation rate indicators are always lagging indicators so 
six years out hopefully these institutions have really done 
some stuff in the past six years.... Sometimes retention data 
for us is more important because it’s something that’s hap- 
pening this year.” 


Lack of accreditor and institutional resources. Many 
accreditors cited a lack of financial, technical, and human 
capital resources as a barrier to more robust data collec- 
tion and use, which lead to difficulties in determining which 
metrics to prioritize. One accreditor said, “Our data system 
is very antiquated. So, we're not able to do in-house the level 
of analysis that we would like.” Similarly, on the institutional 
side, many institutions with fewer resources lack in-house 
institutional research departments or dedicated staff avail- 
able to collect, evaluate, and provide numerous outcome 
indicators on an annual or ongoing basis. A commission 
representative said, “The wealthier institutions have greater 
capacity to collect and use data than the ones that are 
struggling which are more likely to have the students who 
are struggling as well. So, there is a kind of a rich-get-richer 
phenomenon here.” 


However, accreditors generally said they are increasing 
their research and analytics personnel and devoting more 
resources to data-related efforts. As mentioned above, sev- 
eral accreditors noted ongoing efforts to adopt a more mod- 
ern system, with Salesforce often mentioned as an example, 
to leverage analytical capabilities to facilitate data-use and 
data sharing among accreditors. 


Similarly, in our convening several commission leaders 
expressed incredulity that a lack of accreditor and institu- 
tional resources should prevent improved data collection 
and use. Common responses we heard included, “get started 
with the data and capacity you already have,” “leverage tech- 
nology to lower staff costs,” and consider working together 
to jointly request National Student Clearinghouse data at a 
group rate, lowering per-accreditor costs of accessing the 
dataset. 


Lack of common taxonomies, definitions, and processes. 
The range and divergence of, for example, the measure of 
a graduation rate across institutions, accreditors, states, 
and ED makes baselines, comparisons, and generalizable 
findings more difficult—and, in turn, makes standardized 
processes such as data requests from states much more 
time-consuming and difficult than if standard data were 
commonly available. For example, programmatic accred- 
itors seeking state licensure passage rates across several 
states often must engage in a process that is much more 
complex than simply submitting a request for a single data 
file on passage rates. Since the method of requesting var- 
ies state by state there is no common data set or file type, 
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and many state offices are understaffed. One accreditor 
reported both frustration and progress: “Trying to get infor- 
mation from states and licensing organizations about larger 
student performance as compared to one institution is a 
challenge, but it’s getting better.” 


Potential to use data-driven assessment in reductive, puni- 
tive ways. Some claim that the increasing transparency of 
decision-making—and the institution-level or program-level 
data that led to such determinations—could be used to pun- 
ish institutions or accreditors themselves. One commission 
representative remarked that “Something | found troubling 
and challenging is how data in the last five to eight years 
that was put out in public domain turned out to be weap- 
onized in a way. A way to claim misrepresentation, a way to 
say accreditors were doing a terrible job of monitoring what 
their institutions do, when really it was a good faith effort to 
try to start sharing.” 


Similarly, others argued that increased transparency would 
undermine the trust that is important in the peer review 
process: “We're not out there to bust anybody. We’re not out 
there to degrade or... we want programs to be successful. 
We want them to be healthy. We want to help them get there. 
So, it has to be a really trusting relationship. When you have 
that kind of situation, we can’t just turn around and make 
the data available to everyone.” Other explanations include 
the argument that open admissions policies prevent insti- 
tutions from exercising influence over the success of their 
students and that institutions are “doing their best” and 
should not be forced to address outcomes like graduation, 
debt, or employment that some accreditors claim are out- 
side of institutional control or influence. 


Certainly, there are ways in which accreditors walk a fine 
line between wanting to create trusting, honest relation- 
ships and wanting to speak frankly about institutions that 
fail to deliver adequately to their students—even, in some 
cases, due to indifference or negligence. One commission 
representative acknowledged, “We're not well-equipped to 
deal with bad actors. We are used to dealing with idealistic, 
highly committed institutions who are doing their best for 
their student population. Who can get better by learning 
from each other. So, | understand the need from a policy 
point of view to actually arm us also with ways to identify the 
bad actors that may be entering this field for other reasons.” 


While these points reflect the sometimes-difficult realities 
of effectively implementing data and improving the quality 
of higher education, the fact that other accreditors—and 
institutions themselves—have addressed some or even 
all of these issues, shows that more can be done despite 
challenges. 


RECOMMENDATIONS 


Incorporating thoughtful data-use into institutional and 
accreditor policies and practices can have tangible, positive 
effects for students, especially students from low-income 
backgrounds and students of color. Disaggregating data 
should encourage more complete conversations about 
completion by spotlighting equity gaps to spur deeper 
evaluation of the systemic inequities that may be embed- 
ded in institutional policies and practices. Disaggregating 
data can also advance specific remedies, based on prob- 
lems identified through data and progress or challenges 
monitored through data, informing improvement efforts. 
For example, institutions like Florida State University, 
San Diego State University, the University of Wisconsin- 
Eau Claire, and Georgia State University have all improved 
student outcomes through a deliberate focus on data.”! 
Furthermore, by systemically incorporating data into insti- 
tutional improvement efforts, accreditors can signal to poli- 
cymakers that they recognize their responsibility to address 
low-performing institutions in evidence-driven ways. 


While these recommendations are most critical for accred- 
itors serving in a Title lV gatekeeping role, because of their 
critical role in protecting students’ and taxpayers’ interests, 
particularly in cases where persistently low-performing 
institutions and programs fail to live up to their obligations, 
they are also relevant to the work of accreditors who do not 
serve such a gatekeeping role. These accreditors are arbi- 
ters of quality and continuous improvement in their own 
right, and these recommendations will benefit the students 
enrolled at the institutions and programs they oversee. 


Recommendation: Embed data-use into routine practice. 
Accreditors should use data to explicitly inform 
their focus and conclusions by routinely leveraging 
existing federal data sources and, when necessary, 
requiring reporting of additional quantitative student 
outcome data directly from their institutions. 


Each institutional accreditor has the difficult task of over- 
seeing dozens or sometimes hundreds of institutions, each 
of which has a distinct profile and mission. Although this can 
create challenges, it does not mean that empirical outcomes 
data cannot be used to compare similarly situated institu- 
tions, provide early warnings before institutions come up for 
review, or inform the site review process. Indeed, during our 
convening of accreditors, most commission leaders agreed 
that even if current data are not yet optimal, they should 
nevertheless work with their institutions to evaluate the 
data that are available while working to make improvements 
on other data availability. 
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Accreditors should integrate data into their review pro- 
cesses in the following ways: 


Routinely collect, monitor, and act on multiple measures. 
Accreditors should collect data in ways that align with field 
efforts, as summarized in IHEP’s “Toward Convergence” met- 
rics framework—at least every two years for all institutions 
and programs, and annually for institutions or programs that 
manifest worrying quantitative results on student outcome 
measures. Many accreditors have already recognized the 
importance of using multiple measures, as indicated by the 


conclusions in the C-RAC report.” 


It may be appropriate for different accreditors to emphasize 
different metrics based on mission, or even to emphasize 
different metrics for different types of institutions if the 
accreditor oversees a heterogeneous pool. But it is critical 
that accreditors set out clear expectations for their institu- 
tions so that the public has an assurance that the institution 
they attend (and for which they are receiving Pell Grants and 
student loans) meets a minimum level of quality. 


Wherever possible, accreditors should rely on federal 
data sources, such as IPEDS, the College Scorecard, and 
the Federal Student Aid Data Center. Using common data 
sources, especially those freely available from ED, maxi- 
mizes consistency of metrics and minimizes burden on both 
institutions and accreditors. Accreditors should also con- 
sider alternate data sources, such as the National Student 
Clearinghouse or state data dashboards, and only in those 
instances where critical data are unavailable elsewhere 
should accreditors collect data directly from institutions— 
to minimize duplication of effort on the part of both insti- 
tutions and accreditors. Because accreditors are better 
equipped to drive consequences with their institutions, as 
opposed to the federal government's single, binary option 
of allowing or cutting off federal aid eligibility, their great- 
est value-add lies in interpreting data and using it to drive 
conversation with institutions and programs, rather than 
building extensive data warehouses requiring specialized 
analytic capabilities. 


Disaggregate student outcomes by at least race/ethnic- 
ity and income/Pell status. As discussed in more detail in 
Recommendation 2, accreditors should not presume that 
aggregated performance metrics suffice to surface the 
quality of the institutions they oversee. As has been repeat- 
edly demonstrated, many institutions and programs have 
failed to provide the educational resources, supports, and 
inclusive environment necessary to serve students histor- 
ically underserved by higher education. It is critical that 
accreditors collect outcome metrics disaggregated by 
income and race to determine how institutions are serving 
low-income students and students of color and then act 


upon these disaggregated results to promote a more equi- 
table higher education system. As we discuss more fully 
below, we believe improvements in federal data sources 
should, in the longer term, ease the burden on accreditors 
of collecting such information. 


Until then, accreditors will need to collect some of these 
outcomes data by requesting information from institutions 
that is not readily available from other sources. Given the 
importance of evaluating outcomes for underserved stu- 
dents, this additional collection is worthwhile. To simplify 
this process in the long term, accreditors should add their 
voices to calls for improved data collections at the federal 
level so disaggregated results can be published nationwide 
in amore comprehensive way. 


Prioritize accreditor resources and actions based on stu- 
dent outcome benchmarks. As EducationCounsel has 
previously explored, our system of higher education would 
benefit significantly if accreditors provided oversight and 
quality assurance based on a risk-informed framework—as 
not all institutions and programs require the same level of 
resources and oversight. As EducationCounsel has com- 
mented, “instead of requiring the institution to comply with 
every requirement and regulation, institutions would instead 
comply with some baseline rules. Accreditor assessments 
that reveal issues would give rise to additional action on the 
part of regulators and/or their non-governmental partners 
that ranges from requiring more information or participa- 
tionin technical assistance programs, mandating corrective 
action, or—at worst—applying sanctions.” Performance on 
outcomes metrics across institutions would enable accred- 
itors to prioritize their resources to institutions showing the 
most risk to taxpayers and students.” 


Ideally, these efforts will lead to review cycles that incor- 
porate data proactively at every stage of the regular insti- 
tutional review process (this typically occurs every several 
years, although the specific number of years varies by 
accreditor), allowing accreditors and their peer review 
teams to take a targeted approach. Unfortunately, some 
accreditors reverse this approach—reviewing quantitative 
outcome metrics only if a review team perceives a lack 
of quality. For example, one commission representative 
described the process like this: “If we have a team that 
goes in and they see that the quality isn’t looking good, in 
what's going onin the classroom and all that, they're likely to 
take a peek—they’re not required to—but they'll take a peek 
at the assurance system to Say, ‘oh, well it's showing in the 
graduation rates because there's only a three percent grad- 


m 


uation rate.” If, instead, a team were armed with empirical 
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IT IS CRITICAL THAT ACCREDITORS COLLECT 
OUTCOME METRICS DISAGGREGATED BY INCOME 
AND RACE TO DETERMINE HOW INSTITUTIONS 
ARE SERVING LOW-INCOME STUDENTS AND 
STUDENTS OF COLOR AND THEN ACT UPON 
THESE DISAGGREGATED RESULTS TO PROMOTE A 
MORE EQUITABLE HIGHER EDUCATION SYSTEM. 


outcome metrics on the front end, that knowledge could 
trigger a much different review process, set of inquiries, and 
required responses. 


Accreditors have many touch points with institutions each 
year, and accreditors do conduct oversight outside the reg- 
ular review cycle. To target these efforts and objectively 
determine where deeper intervention is necessary, accred- 
itors should establish minimum standards below which spe- 
cific consequences apply, and graduated thresholds that 
encourage continuous improvement. Many commission 
leaders reflected on the importance of nuance in describ- 
ing “consequences.” ACCSC, for example, sets bright-line 
thresholds to measure performance; nevertheless, failure 
ona particular metric in a given year will not trigger loss of 
accreditation, but will instead trigger a deeper inquiry into 
the reasons why such performance data are below accept- 
able thresholds. 


This approach is appropriate as an initial response. The 
first step is to start a conversation about the specific ways 
an institution or program can improve, rather than issuing 
an overly reactive knee-jerk response to a single year of 
poor performance. However, continued performance below 
benchmark thresholds—absent mitigating circumstances, 
which can be demonstrated through supplemental data 
(quantitative and qualitative)—must be met with increasingly 
serious responses, including targeting additional accreditor 
resources and site visits, instituting enrollment caps, and in 
cases where failure to improve continues, revoking approval 
for low-performing programs and revoking accreditation.” 


Indeed, ACCSC, who has used performance thresholds for 
more than two decades, confirmed that increasing conse- 
quences were a necessary component of their oversight 
functions to ensure that member institutions take seriously 
initial warnings to improve. As one commission represen- 
tative recognized, “If it’s a more systemic kind of thing and 
multiple programs have outcome issues and institutions 
aren't able to work through those outcomes issues, unable 
to recognize those outcomes issues, aren't able to mitigate 
or correct those outcomes issues, then that rises to the 


level of institutional action of revoking or withdrawing an 
institution’s accreditation. But if it’s just a single program 
that they're trying to work through and again, the institution 
is being a bit dense about coming to this conclusion on their 
own, our commission is much more apt to say we're ceasing 
approval of this program.” 


This type of ongoing, tiered performance monitoring would: 


e Enable accreditors to proactively identify institutions or 
programs that may present particular risk to students and 
taxpayers; 


e Allow accreditors to prioritize staff and resources to 
the institutions that need it most—including extending 
the review cycle for high performers, freeing up more 
resources to focus on poor performers; 


e Help identify institutions that have improved over time 
or that serve low-income students and students of color 
particularly well; and 


e Facilitate sharing of lessons learned across similar 
institutions. 


Recently, regional accreditors signaled their ability and will- 
ingness to experiment with this type of approach by agree- 
ing on a common graduation rate metric and associated 
thresholds. As part of a C-RAC exercise, accreditors took 
a closer look at four-year institutions that had graduation 
rates at or below 25 percent and two-year institutions that 
had graduation rates at or below 15 percent.” During our 
convening, commission leaders also expressed some will- 
ingness to have “heightened attention” based on zones or 
ranges of concerning outcomes data, rather than clear cut- 
offs. Given the benefits of such approaches, accreditors 
should build on and expand these initial efforts. 


Recommendation: Emphasize equity. Accreditors should 
make equity a higher priority by requiring disaggre- 
gation of quantitative outcome metrics by race and 
income. 


To advance equitable opportunities for students from 
low-income backgrounds and students of color, accreditors 
should require disaggregation of institutional and program- 
matic outcome data. Despite increased attention to overall 
metrics like graduation rates, accreditors collect sparse 
information, if any at all, on how institutions are serving his- 
torically underserved students (see Finding 4). 


Data reporting requirements signal priorities. When 
accreditors require—or don’t require—particular data ele- 
ments, they send a powerful message about the impor- 
tance of those data to institutions and programs. Also, by 


18 Informing Improvement | 


collecting disaggregated data, they can better discern 
trends otherwise obscured in topline results. 


The biggest barriers among accreditors to implementing 
this recommendation appears to be that some do not see 
this role as within their purview, and that some worry about 
misuse of disaggregated data. As we discussed in detail in 
Finding 4, we believe that accreditors have both a responsi- 
bility to undertake efforts that improve equity in the institu- 
tions they oversee, and that doing so will provide significant 
benefits to our system of higher education. Furthermore, 
by using disaggregated data, accreditors can help craft 
the narrative about what those results mean and how they 
should be used to improve outcomes for students. 


Finding 4 also emphasizes that disaggregation is, by itself, 
insufficient to address systemic racial and income inequities 
in higher education. However, disaggregated data reporting 
is a necessary prerequisite to understanding where and to 
what extent such disparities still occur. Only by being able 
to first ascertain where institutions and programs are not 
serving students of color and students from low-income 
backgrounds can accreditors begin to target resources 
and push institutions to devote more attention, effort, and 
spending to address poor institutional performance. Even 
those institutions that appear to perform well on overall 
metrics may demonstrate equity shortcomings when disag- 
gregated data are available. Also, disaggregated data might 
reveal that an institution has made improvements toward 
closing the opportunity and achievement gap between stu- 
dents of color and white students. 


Recommendation: Increase transparency about data-use 
practices. Accreditors should build on the progress 
established in the C-RAC report to improve data col- 
lection, and to better explain to the public, including 
policymakers, what data they collect, how they col- 
lect that data, and how they use that data in their 
review processes. 


Improving data collection. Accreditors should continue 
their existing, proactive efforts to improve the quality of 
data collection and its use (similar to the valuable work that 
occurred as part of the C-RAC graduation rate exercise) in 
two respects: (1) replicate and routinize the experiment and 
include national and programmatic accreditors where pos- 
sible and (2) do so with more data elements, disaggregated 
by race/ethnicity and income. 


» Replicate and routinize the C-RAC experiment across 
all types of accreditors. Routinizing the graduation rate 
exercise would be valuable because each new iteration 
would bring new lessons and iterative improvements that 


can be disseminated to partner accreditors, who will in 
turn share best practices with their member institutions. 
Furthermore, it would elevate the collective importance 
of the outcome metrics and encourage future collabora- 
tion across institutions and accreditors. 


Indeed, the C-RAC report itself acknowledged the impor- 
tance of continuing these efforts: “Regional accreditors 
will continue to monitor these efforts and to provide pol- 
icymakers with information about what they are learning 
from monitoring and research.” Accreditors have an import- 
ant opportunity to help policymakers understand what is 
collected, how it is used, and how that translates to more 
effective quality assurance, which will strengthen trust in 
our systems of accreditation and higher education. 


« Expand to more data elements, disaggregated by race/ 
ethnicity and income. In addition to continuing the 
practice of evaluating institutional graduation rates, 
accreditors should expand to address multiple, disaggre- 
gated data points. These should include improved grad- 
uation rates (i.e. IPEDS Outcome Measures), retention 
rates, course completion rates, workforce outcomes, and 
licensure passage rates, as applicable. By expanding the 
project to cover other types of measures, accreditors can 
avoid gaming of a single metric or painting an incomplete 
picture of institutional performance, including, most seri- 
ously, risking overlooking other problems. 


We measure what we value. When accreditors ask questions 
of institutions’ current practices, and request data tracking 
the success and challenges of those practices, they signal 
to those institutions that these subjects are important to 
their accreditors. In addition, as accreditors conduct reg- 
ular, Ongoing outreach on a broader set of data measures, 
they send a powerful framing message to institutions about 
the importance of making data-driven decisions. This mes- 
saging also signals the importance of institutional focus on 
equitable educational outcomes for students of color and 
low-income students. 


Building trust and understanding by _ transparently 
explaining what data accreditors use, how they collect it, 
and how they use it to make decisions. As noted in Finding 
3, accreditors often claimed to use data, but only offered 
vague descriptions of how they used data. To avoid mis- 
understandings and raise the level of policymaker trust, 
accreditors should make commitments to publicly describe, 
in detail, what data they are using, how they collect that data 
(the specific steps they are taking with respect to their mem- 
ber institutions), and how they will use the tools—especially 
student outcome data—at their disposal. In increasing this 
transparency, accreditors should be clear about their own 
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measures of quality and their efforts to work together to 
develop shared metrics and thresholds, as they did in the 
C-RAC report. 


Enhanced transparency into how decisions are made is 
equally important in restoring trust in the accreditation sys- 
tem. As we have seen in numerous other higher education 
contexts—and as policymakers’ increased focus on how to 
improve accreditation has made clear—the public will not 
simply buy into the notion that accreditors are effectively 
assuring the quality and improvement of the nation’s col- 
leges and universities without evidence. 


cc WE MEASURE WHAT WE VALUE. 


Many of the accreditors we interviewed agreed that policy- 
makers are disconnected from the accreditation process 
and do not understand all the tasks and responsibilities 
involved. For instance, during our convening, several accred- 
itors suggested that policymakers and policy organizations 
could better understand the day-to-day responsibilities 
of accreditors, and demonstrate a commitment to under- 
standing those responsibilities by, for example, observing 
a site visit or coming to speak with accreditor representa- 
tives. This feeling of misunderstanding leads to accreditors’ 
sense that they are being forced to undertake responsibili- 
ties outside their mission, expertise, and capacity. However, 
without transparency from accreditors on the part of the 
process they use, including when specific actions are taken 
and why, it is difficult for policymakers or the public at large 
to get inside the “black box’ of the accreditation process. 


Providing more detail on how data are used in decisions, 
including evidence of the connection between outcomes 
and action, will provide important assurances to policy- 
makers and the public that accreditors understand and take 
their role seriously as arbiters of quality. It will also provide 
value to accreditors—by demonstrating to policymakers and 
the public that accreditors themselves see value in integrat- 
ing data into the quality assurance system, thus bolstering 
confidence in the oversight abilities of accreditors. 


As institutions are using data to drive improvements in student outcomes and educational 
equity, accreditors also have a role to play in promoting student success for all students, 
especially low-income students and students of color. However, while accreditors are 
collecting some outcomes data, most are not yet assessing a wide spectrum of quantitative 
outcomes metrics; very few disaggregate performance data by race or income; and data- 


use practices do not seem to be thoroughly embedded in their day-to-day activities. 


Recent years have shown progress in accreditors’ data-use practices, as evidenced by the 
C-RAC graduation-rate initiative and the case studies of individual accreditor initiatives 
profiled in this report. Accreditors now have an opportunity to build on this progress by 
connecting quantitative outcomes metrics to their review processes and decisions more 
holistically, disaggregating metrics by race and income, and better explaining the process 
by which decisions are made. If policymakers continue to perceive a failure on the part of 
accreditors to reflect outcomes data in their quality assurance processes, they may choose 


to impose such requirements in law, 

DATA ALONE CANNOT DRIVE INSTITUTIONAL 

IMPROVEMENT, BUT A DATA-DRIVEN CULTURE 

barriers and realities accreditors face. OF INQUIRY—LED BY SEASONED ACCREDITORS 
AND HIGHER EDUCATION PROFESSIONALS— 


: , ee CAN INCITE THE TYPE OF IMPROVEMENTS 
Accreditors hold the weighty responsibility THAT TODAY'S STUDENTS DESERVE 


perhaps in ways that fail to reflect the 


of protecting and promoting quality 

educational opportunities for postsecondary students of all backgrounds. To fully deliver on 
this charge, they must rely more heavily on quantitative student outcome data as a tool to 
uncover problem areas, racial and socioeconomic inequities, and examples of institutional 
and programmatic successes. Data alone cannot drive institutional improvement, but a data- 
driven culture of inquiry—led by seasoned accreditors and higher education professionals— 


can incite the type of improvements that today’s students deserve. 


20 Informing Improvement | 


APPENDIX A 

Participating Commission Leaders 

Barbara Brittingham, President, New England Commission of Higher 

Barbara Gellman-Danley, President, and Patricia O’Brien, Senior Vice President, Higher Learning Commission 
Leah Matthews, Executive Director, Distance Education Accrediting Commission 

Michale McComis, Executive Director and CEO, Accrediting Commission of Career Schools and Colleges 
Michael Milligan, Executive Director and CEO, ABET 


Marlene Moore, Former President, and Sonny Ramaswamy, President and CEO, Northwest Commission on Colleges and 
Universities 


Marsal Stoll, CEO, Accreditation Commission for Education in Nursing 
Jamienne Studley, President and CEO, and Henry Hernandez, Former ClO, WASC Senior College and University Commission 


Belle Wheelan, President, and Alexei Matveev, Director of Training and Research, Southern Association of Colleges and 
Schools Commission on Colleges 


Richard Winn, President, Accrediting Commission for Community and Junior Colleges 


APPENDIX B 

Abbreviations and Acronyms 

Absolute Graduation Rate (AGR) 

Accrediting Commission for Community and Junior Colleges in the Western Association of Schools and Colleges (ACCUC) 
Accrediting Commission of Career Schools and Colleges (ACCSC) 

Accreditation Commission for Education in Nursing (ACEN) 

Council for Higher Education Accreditation (CHEA) 

Council of Regional Accrediting Commissions (C-RAC) 

Graduation Rate Dashboard (GRD) 

Higher Learning Commission (HLC) 

Institute for Higher Education Policy (IHEP) 

Integrated Postsecondary Education Data System (IPEDS) 

IPEDS Outcome Measures (OM) 

National Center for Education Statistics (NCES) 

New England Commission of Higher Education (NECHE) 

The Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) 
United States Department of Education(ED) 

Unit Redemption Rate (URR) 

WASC Senior College and University Commission (WSCUC) 
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